Magnitudes of Relevance: Relevance Judgements, Magnitude Estimation, and Crowdsourcing

Magnitude estimation is a psychophysical scaling technique where the intensity of a stimulus is rated by the assignment of a number. We report on a preliminary investigation on using magnitude estimation for gathering document relevance judgements, as commonly used in test collectionbased evaluation of information retrieval systems. Unlike classical binary or ordinal relevance scales, magnitude estimation leads to a ratio scale of measurement, more suitable for statistical analysis and potentially allowing a more precise measurement of relevance. By performing a crowdsourcing experiment, we show that magnitude estimation relevance judgements are consistent with ordinal relevance ones; we study the difference of using a bounded or an unbounded scale; we show that magnitude estimation can be a useful tool to understand the perceived relevance when using an ordinal scale; and we investigate document presentation order effects.